Context. Observations at sub-millimeter and mm wavelengths will in the near future be able to resolve the radial dependence of the mm spectral slope in circumstellar disks with a resolution of around a few AU at the distance of the closest star-forming regions. Aims. We aim to constrain physical models of grain growth and fragmentation by a large sample of (sub-)mm observations of disks around pre-main sequence stars in the Taurus-Auriga and Ophiuchus star-forming regions. Methods. State-of-the-art coagulation/fragmentation and disk-structure codes are coupled to produce steady-state grain size distributions and to predict the spectral slopes at (sub-)mm wavelengths. Results. This work presents the first calculations predicting the mm spectral slope based on a physical model of grain growth. Our models can quite naturally reproduce the observed mm-slopes, but a simultaneous match to the observed range of flux levels can only be reached by a reduction of the dust mass by a factor of a few up to about 30 while keeping the gas mass of the disk the same. This dust reduction can either be due to radial drift at a reduced rate or during an earlier evolutionary time (otherwise the predicted fluxes would become too low) or due to efficient conversion of dust into larger, unseen bodies.